Skip to content

fix(relay): make a cold relayed capsule fetch reachable in one command - #334

Merged
MichaelTaylor3d merged 23 commits into
mainfrom
loop/3128-relay-batch
Aug 24, 2026
Merged

fix(relay): make a cold relayed capsule fetch reachable in one command#334
MichaelTaylor3d merged 23 commits into
mainfrom
loop/3128-relay-batch

Conversation

@MichaelTaylor3d

@MichaelTaylor3d MichaelTaylor3d commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

DO NOT MERGE — gate round not yet run.

dig-node's whole remaining share of dig_ecosystem#3128, as one batch (CLAUDE.md §1.4 batch-per-repo).

Closes #322
Closes #332
Closes #333


The shape decision for #333, and its argument

A blocking ask must be capped tightly; a non-blocking one need not be. The 65 s descriptor ladder
is not too short — it is the right bound for the operation it was built for, and the operation was
wrong.

The hop held the requestor's stream open for the whole duration of a third-party 135 MB transfer, so
the cost of waiting was a held stream on both ends. That is exactly the cost a wall-clock cap exists to
bound, which is why the cap is correct and why the capability was unreachable behind it.

Neither obvious fix works:

  • Widening the ladder lets an unanswerable holder hold a slot for minutes — the amplification the
    two-phase proxy and proxy_rate_limiter exist to bound.
  • Deriving a deadline from capsule size is circular: the size arrives in the descriptor, which is
    the thing being waited for.

So the fix decouples the two waits rather than lengthening one:

  1. Hop side. relay_capsule returns RelayStatus { Landed | Pending { staged_bytes } | Refused }.
    The pull is spawned in the background and the ask waits only a 3 s grace — comfortably inside the
    requestor's first 5 s rung, so a capsule that lands quickly is still answered in one round trip and
    the case that already worked is unchanged. One that does not is answered ContentMissInconclusive
    (-32017) with error.data.relay_staged_bytes, and the pull keeps running.
  2. Requestor side. On that answer the requestor stops using a wall clock and waits on PROGRESS:
    re-ask every 10 s, continue while the staged count strictly advances, abandon after a 60 s stall
    window with no advance, and abandon at a 30 min ceiling regardless.

Once the ask is non-blocking, the instrument changes. Each further poll costs one small round trip
instead of a held stream, so "how long may I wait" stops being a resource question and becomes a
liveness question. Forward progress needs no size; a wall clock needs a size the requestor cannot have.

Why the ceiling is not optional (NC-12). relay_staged_bytes is a HOP'S CLAIM about itself. A
stall window cannot catch a hostile hop that fabricates a counter rising forever, so the ceiling is
what makes the worst case finite: a lying hop can waste one pull's time from one peer, and no more.
The claim decides only how long to wait — never whether a byte is accepted. The capsule is still
verified against the chain-anchored root exactly as a direct holder's would be, so a hop that
fabricates its way through a wait still cannot produce content that passes.

Backwards compatible by construction. The code is the taxonomy's existing inconclusive-miss code,
not a new number (SYSTEM.md: the taxonomy is owned by dig-rpc-protocol and adopted, never
restated — this repo has paid for restating it twice). A requestor that ignores relay_staged_bytes
sees an ordinary inconclusive miss and retries later, which is precisely what #333's field observers
were doing by hand.

The diff, per ticket

#333module_relay.rs (RelayStatus, the 3 s grace, background spawn), module_reshare.rs
(spawn_relayed_capsule_warm, holds, staged_bytes, await_landing), lib.rs
(relay_pending_err, RELAY_PROGRESS_FIELD, both serve sites), peer.rs +
module_serve.rs (module_relay_pending_frame for the streaming site), module_transport.rs
(DescriptorAnswer, wait_for_relayed_descriptor, the three bounds).

#322module_transport.rs: LadderEnd now distinguishes Refused (the peer ANSWERED, and the
answer was no) from Exhausted (it never answered). A refusal escalates within the same invocation;
an exhausted ladder does not, because a peer that could not answer a plain ask will not answer a relay
ask and escalating would double the invocation's bound for nothing.

#332module_reshare.rs: StagedPull holds the staged bytes and the resume checkpoint as ONE
value with one erase, so no failure path can remove half a partial. Discard erases both;
Preserve touches neither.

Decisions the coordinator asked me to state

StagingDisposition is KEPT, not removed. With dig-download 0.19.2's demotion wipe gone, the
disposition is not belt-and-braces — it is now the ONLY thing that erases a partial attributable to a
proven-false descriptor. The lower layer deliberately PRESERVES on demotion; something still has to
decide that a verification failure is different, and that decision has exactly one home.

dig-download is bumped to 0.19.2 in the lock (lock-only; the "0.19" caret already admits it).
Verified from the index: vers=0.19.2, yanked=false, checksum 02a51e9858830dcc…, and the
non-dev dependency set is byte-identical to 0.19.1, so the single [[package]] block swap is the whole
correct change. cargo metadata --locked passes.

No other dig-*/chia-* bump is takeable here (§2.4b), and this is measured, not assumed. Every
latest release reaches chia 0.36 — dig-download 0.20.3, dig-nat 0.20.0 and dig-dht 0.12.1 all
require dig-tls ^0.4 (chia-bls ^0.36.1), and dig-peer 0.12.0 requires chia-protocol ^0.36 — while
this crate is on chia-protocol 0.26 / chia-bls 0.26 / chia-wallet-sdk 0.30. Taking any one of them
would put two chia lines inside one crate, which is the internally-split state §2.4b exists to prevent
and which shipped twice on 2026-08-22. The uplift is the chia-0.36 cascade (dig_ecosystem#3152), not
this PR.

Two false manifest comments corrected (dig-node-core/Cargo.toml, dig-wallet/Cargo.toml). They
said the chia-peer unpin awaited a release built against dig-chainsource-interface 0.2. Measured on
the index: no such release was ever neededchia-peer 0.2.0 skipped dci 0.2 and went straight to
dci ^0.3 + chia-protocol ^0.36.1, where chia-query 0.15.0 also sits, so the two
ChainSourceProvider traits already unify again. They unify at CHIA 0.36, so the real blocker is the
cascade above. A stale blocker is worse than none, because it stops people looking.

Blast radius (§2.0)

gitnexus indexed per-worktree (dn-3128, 114 MB) and queried; paired with ripgrep, because gitnexus
under-counted here — it found 1 caller of relay_capsule where grep found 3.

symbol risk callers
relay_capsule LOW 3 — peer.rs::stream_module_range, lib.rs::get_module_info, lib.rs::fetch_module_range_frame. All three updated.
discard_stagingStagedPull::erase LOW 2 direct (StagingDisposition::apply, warm_with_config)
descriptor_over LOW 1 (get_module_info)
ask_within_deadlines MEDIUM 7 direct, 6 of them its own tests

No public API of dig-node-core changes shape; every edited symbol is pub(crate) or private, and the
wire change is one additive error.data field.

Verification

Revert-proofs, each run alone, naming the assertion that fired — see the PR comments for the table.

@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

Progress — resume-ready

Branch loop/3128-relay-batch · HEAD 2a27fd3 (pushed) · DRAFT, do not merge — gate round not yet run.

Done

The finding that mattered, and what it changed

The first revert-proof pass ran a revert called no-wait — removing the relay wait from
get_module_info — and nothing failed. The four #333 tests were driving
wait_for_relayed_descriptor directly, so they proved the wait was correct while saying nothing
about whether the production path still reached it. That is the same class of defect as #328
shipping broken twice: the right property, asserted against a fixture that cannot exercise the real
path.

The fix was structural rather than a bigger test. The three steps — ask, escalate, wait — are one
decision about a holder that did not simply hand over the descriptor, so they now live in ONE
function, descriptor_via_rounds, which get_module_info calls and which every test drives. A
helper extracted purely for testability is a helper a call site can quietly stop using.

In progress

Re-running the five revert-proofs against the folded shape, individually rather than as a batch —
the no-ceiling revert makes the wait unbounded, so it hangs rather than failing, and a batch run
cannot attribute that to a single test.

Next action

  1. cargo test -p dig-node-core --lib seams::dig_peer::module_transport — confirm 21 green.
  2. Five reverts via scratchpad/revert.py, one at a time, each naming the assertion that fired.
  3. Version bump, CHANGELOG, SPEC/docs sweep, §2.4b dep pass, then the gate.

@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

Revert-proof evidence

Each revert applied ALONE to a committed tree, run alone, restored afterward. Committing first is not
optional here — git checkout <path> is destructive, and a background proof script's trailing
checkout clobbered an uncommitted edit of mine twice during this work.

revert tests that failed assertion that fired
no-wait — a RelayPending answer returns NoAnswer instead of waiting 4 — all of the #333 group a hop that is moving bytes must be waited for: Err(NoAnswer), and three left: NoAnswer
no-ceiling — drop elapsed() >= RELAY_MAX_WAIT 1a_hop_that_fabricates_endless_progress_is_bounded_by_the_ceiling the wait MUST end at the ceiling; an unbounded wait is the NC-12 defect itself: Elapsed(())
no-stall — drop the stall branch 1a_frozen_relay_is_abandoned_after_the_stall_window the Stalled equality at :1145
no-escalation — one round per invocation 1a_cold_first_invocation_spends_a_plain_round_then_escalates the phase sequence, [false] vs [false, true]
escalate-always — escalate whenever no descriptor came back 2an_unresponsive_holder_is_not_escalated AND the_production_get_module_info_climbs_the_whole_ladder the ladder-total equality: 130 s vs 65 s, through the real method, in 60 s of wall clock
#332 Discard no longer clears the checkpoint 1a_verification_failure_still_discards_what_it_staged a discarded partial must leave no resume checkpoint behind
#332 Preserve also clears (the over-broad direction) 1a_severed_warm_resumes_from_its_partial_instead_of_refetching_the_capsule the resumed byte count at :1567

The one that mattered: no-wait fired NOTHING on the first pass

The four #333 tests were driving wait_for_relayed_descriptor directly. They proved the wait was
correct and said nothing about whether get_module_info still reached it — the right property against
a fixture that cannot exercise the real path, which is the shape that let #328 ship broken twice.

Why the fixture missed: the wait had been extracted purely for testability, so the tests and the
call site were two independent things and only one of them was pinned.

What changed: the three steps — ask, escalate, wait — are ONE decision about a holder that did not
simply hand over the descriptor, so they now live in one function, descriptor_via_rounds, which
get_module_info calls and which every test drives. Re-run: no-wait fires on all four, naming
Err(NoAnswer) where a descriptor was expected.

The general lesson, since it caught two separate things here: a helper extracted for testability is
a helper the call site can quietly stop using.
Prefer folding the decision into one function over
adding a test that asserts the seam is still wired.

Two other things the proofs caught, which are worth recording

  1. A killed proof batch silently un-shipped a security bound. The first no-ceiling run hung (an
    unbounded wait spins virtual time forever); the batch was killed at its timeout before its restoring
    git checkout ran, and the emergency "commit everything now" after a cap swept the revert into
    2a27fd3. The ceiling was missing from main-bound code for four commits. Restored in 4da7ecc,
    and found only because the proof discipline sent me back to look at the pattern.
  2. A test that HANGS on regression is a landmine, not a proof. The ceiling revert burned 659
    CPU-seconds and reported nothing attributable. The test is now bounded by
    tokio::time::timeout(RELAY_MAX_WAIT * 2, …), so the same regression now fails in 0.01 s with a
    sentence naming the defect.

@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

Implementation complete — handing back to the review gate

Branch loop/3128-relay-batch · HEAD 5305989 (rebased onto 8f11afa, pushed) ·
Still DRAFT. The §2.4a gate round has not been dispatched; this lane does not run it.

Local evidence

gate result
cargo fmt --all -- --check 0
cargo clippy --workspace --all-targets -- -D warnings 0
cargo test --workspace 0 — 968 in dig-node-core lib, ~2,300 across the workspace, 0 failed
revert-proofs 7, each run alone, each naming the assertion that fired (comment above)
dig-download 0.19.2 in the lock cargo metadata --locked passes

CI: every required check green except Test + coverage, still running at the time of writing.
check-merge-preconditions.sh reports BLOCKED [EXIT 1] on that one check alone —
unresolvedReviewThreads=0, and the other four required contexts are present and SUCCESS by name.

The acceptance bar this PR does NOT clear, stated plainly

§2.6's bar is a person watching a .dig arrive over a hop, on real hardware. This lane cannot
produce that evidence
#322, #332 and #333 were all measured on a three-machine A → B → C run, and
I have unit-level proof only.

What that means concretely: the timing constants — a 3 s hop grace, a 10 s poll, a 60 s stall window,
a 30 min ceiling — are argued from the field measurements on the tickets, not confirmed by a repeat of
that run. The hardware acceptance re-run is the remaining work on #333, and it is worth doing
against this branch rather than against v0.145.0, whose lock still resolves dig-download 0.19.1.

One correction to the brief I was given

The two manifest comments were stale, but not in the way reported. The claim was that the chia-peer
unpin awaited a dig-chainsource-interface 0.2 release that had since shipped. Measured on the
index: no such release was ever needed. chia-peer 0.2.0 skipped dci 0.2 entirely and went to
dci ^0.3 + chia-protocol ^0.36.1, where chia-query 0.15.0 also sits — so the two
ChainSourceProvider traits already unify, at chia 0.36. The real blocker is therefore the
chia-0.36 cascade (dig_ecosystem#3152), and unpinning on the strength of the old comment would have
pulled this crate onto a second chia line. Both comments now say that.

@MichaelTaylor3d MichaelTaylor3d left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CHANGES-REQUIRED — correctness gate

Head reviewed: 6e2b6730a772f6fc92b39cd16c8273a666baed0b (resolved from the remote, not from the dispatch).

Recorded as a --comment review: a verdict event from the PR author's identity is a 422 for both APPROVE and REQUEST_CHANGES. The four open inline threads are what bar the merge, via required_conversation_resolution.

What I verified myself, by mutation, in my own worktree

Baseline at head: 968 passed, 0 failed.

Probe Result
no-wait (delete the RelayPending arm) CONFIRMED — all four #333 tests fail. The compiler also reported wait_for_relayed_descriptor as dead code, proving the mutation reached production.
#332 Discard (drop state_store.clear) CONFIRMEDa_verification_failure_still_discards_what_it_staged fails.
#332 Preserve (make Preserve erase) CONFIRMEDa_severed_warm_resumes_from_its_partial_instead_of_refetching_the_capsule fails.
ceiling removed CONFIRMEDa_hop_that_fabricates_endless_progress_is_bounded_by_the_ceiling fails in 0.01 s with Elapsed(()). The tokio::time::timeout bound works; it fails fast, it does not hang.
get_module_info bypasses descriptor_via_rounds DEFECT — 968/968 still pass. See thread 1.

The revert-proofs the PR claims are real. The gap is one level up: they protect the helpers, not the production call path.

The ceiling incident — verified, with one correction

RELAY_MAX_WAIT is present and correct at head (const module_transport.rs:564, the check at :624 before any further sleep, three tests). I traced the history independently: the check was removed in 27c97cd and restored in 203661c. Nothing else from that batch is missing — the only other deletions in 27c97cd are the intended descriptor_with_escalationdescriptor_via_rounds refactor, and all three RelayWaitEnd endings survive in both code and tests.

One correction to the write-up: 2a27fd3 is not an ancestor of this branch. The branch was rewritten; its on-branch twin is 27c97cd. The narrative is otherwise accurate, but a reader chasing 2a27fd3 will not find it in the history.

Verdict on the four constants

  • 3 s grace — sound. Comfortably inside the 5 s first rung, so the small-capsule case stays byte-for-byte one round trip.
  • 10 s poll — sound, with a doc nit: it is described as "one small round trip", but each poll is round(true) = a re-dial plus the full 65 s ladder. Cheap for a hop that answers Pending promptly; a hop that goes silent costs a 65 s ladder before being classed Abandoned. Non-gating.
  • 60 s stall — sound. Correctly sized above the poll interval, and the monotonic-max best makes downward lying useless (it self-stalls at 60 s).
  • 30 min ceiling — right instrument, wrong scope. See thread 2: it bounds one peer, while dig-download's documented worst case is 3 × holders × the transport's per-ask timeout, and this PR raises that term from 65 s to 30 min.

Also confirmed

  • NC-12 core claim holds. staged_bytes is read only for best / last_advance and one tracing::debug!. It never gates acceptance, and the capsule still clears the chain anchor + merkle verification in warm_claiming.
  • §2.4b spot-checked and the lane's claim is correct. dig-nat 0.20.0dig-tls ^0.4; dig-peer 0.12.0chia-protocol ^0.36 + dig-tls ^0.4; dig-dht 0.12.1dig-nat ^0.20, so it is blocked transitively too. dig-peer-selector 0.9.1 is already inside the declared caret. Nothing else is takeable without the chia-0.36 cascade (#3152). The two corrected Cargo.toml blocker comments are an improvement and match the index.
  • capsule_warm_locator_tests — the set-based assertion keeps both halves: an empty set still catches a pull that asked nobody, and a second id still catches reaching past the pool. Correct call, correctly reasoned.
  • #332 / dig-download 0.19.2 composition — the StagingDisposition justification holds. for_failure is the only producer of Discard, erase() is the only path that clears the checkpoint, and I found no second eraser.

Not blocking

.github/** is untouched, so there is no workflow collision with #336 — but Cargo.toml, Cargo.lock and SPEC.md all overlap. Thread 5, self-resolved.

Four threads open. I will re-verify and resolve each against the new head.

Comment thread crates/dig-node-core/src/seams/dig_peer/module_transport.rs Outdated
Comment thread crates/dig-node-core/src/seams/dig_peer/module_transport.rs Outdated
Comment thread crates/dig-node-core/src/seams/dig_peer/module_relay.rs
Comment thread crates/dig-node-core/src/seams/dig_peer/module_reshare.rs
Comment thread Cargo.toml
MichaelTaylor3d and others added 15 commits August 23, 2026 21:53
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…ifest blockers

The ceiling check was removed by a revert-proof batch that was killed before it
could restore the file, and swept into 2a27fd3. It is the NC-12 bound: a hop's
staged byte count is its own claim, so a stall window alone cannot catch a hop
that fabricates rising progress forever.

The two Cargo.toml comments named a blocker that never existed: chia-peer 0.2.0
skipped dig-chainsource-interface 0.2 and went straight to dci ^0.3 with
chia-protocol ^0.36.1, where chia-query 0.15.0 also sits. The real blocker is
the chia-0.36 cascade (dig_ecosystem#3152).

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…ion proof

dig-download 0.19.2 adds an across-round descriptor re-ask (dig-download#37), so a
never-answering holder is now asked MAX_DESCRIPTOR_ATTEMPTS times. The test pinned
the exact ask sequence, which was never its subject -- it proves the warm locator
unions the connected pool. Both halves of that property survive the change to a
set: an empty set still fails, and a second peer still fails.

Co-Authored-By: Claude <noreply@anthropic.com>
…NTS.md

gitnexus analyze rewrites its index name into both files, which are TRACKED, so a
per-worktree index silently lands in the diff.

Co-Authored-By: Claude <noreply@anthropic.com>
…ay waiting per PULL

The ladder, the second-pass escalation and the progress-bounded wait now live in a
private child module. The parent supplies only a policy-free one-shot ask, so a
call site cannot reach the middle of the decision: the gate's bypass -- calling the
inner round directly, which deleted both features with 968 tests green -- no longer
compiles.

RELAY_MAX_WAIT bounds one PEER. The pull asks many, so ten hops each fabricating a
byte of progress per poll stayed inside their ceilings while holding one pull for
hours. RelayWaitBudget charges relay waiting against the CAPSULE, so one honest hop
may still spend the whole budget and liars share it.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…its name

Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d and others added 3 commits August 23, 2026 21:56
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

All four gate findings addressed — re-gate requested

Rebased onto 5fdcf909 (#336). Re-asserted BY VALUE after the rebase, not by "the file changed":
root 0.146.0, dig-node-core 0.55.0, lock dig-download 0.19.2, and both corrected manifest
comments still naming #3152 (4 hits + 1 hit).


Finding 1 — you were right, and my fix had closed the wrong joint

Unifying the three steps made them inseparable from each other while leaving the inner ladder
reachable from the parent. So the call site could still take the first step and skip the rest, which
is what you demonstrated with 968 tests green.

The fix is structural, not another assertion. The ladder, the escalation and the relay wait now
live in a private child module descriptor_ask. ask_within_deadlines is private to it. The parent
supplies only ask_descriptor_once — a dial, a question, a hang-up, carrying no policy at all.
There is no longer anything a call site could use to get a descriptor almost the right way.

Proved the way you broke it. I reproduced your bypass exactly — replaced get_module_info's body
with a direct one-shot ask — and it now fails two production-path tests:

test assertion that fired
the_production_get_module_info_climbs_the_whole_ladder `get_module_info` must be BOUNDED by the ladder; an unbounded ask is the defect: Elapsed(())
a_cold_first_invocation_escalates_through_the_production_transport (new) a refused plain round must be followed by an ESCALATED round inside the same call — 1 round observed, 2 required

The second is new and is the one that pins the call site rather than the helper: a CountingLocator
counts locator consultations, and each descriptor round consults it exactly once, so the round count is
observable through the real method with no peer, no socket and no injected seam.

One thing worth flagging: on the first attempt your bypass made the ladder test hang rather than
fail — with no ladder there is no bound at all, so it waited on the pending locator forever. It is now
wrapped in tokio::time::timeout(ladder * 2, …), so the same regression fails in milliseconds with a
sentence naming the defect. A test that hangs on regression is a landmine, not a proof; that is the
second one on this PR (the ceiling test had the same shape).

Finding 2 — per-pull budget added; the per-hop ceiling untouched

RelayWaitBudget charges relay waiting against the capsule, keyed (store_id, root) — which is
exactly the unit one ModuleDownloader::download call is about, so the key is the pull.
descriptor_for_holder now receives what is left rather than a fresh allowance, and reports its
relay_waited so only actual relay time is charged. One honest hop may still spend the entire budget,
so the large-capsule case you correctly warned about is preserved; ten liars share it.

Three tests, all revert-proved. The load-bearing one is a_hop_gets_only_what_the_pull_has_left,
which catches the shape a half-applied budget takes — a parameter accepted, threaded, then ignored in
favour of the constant. Reverting exactly that line yields:

the wait must end at the budget it was GIVEN, not at the per-hop maximum
  -- elapsed 1800s against a budget of 30s

Also written into SPEC.md §21.1, since a reimplementation reading only the ceiling clause would
rebuild the multiplication.

Finding 3 — WarmRegistry starvation, documented and tested

Recorded at both ends — the spawn site in module_relay.rs and CapsuleWarmer::warm_relayed — that
the cap is global and shared with local warms, and that since the pull became spawned it survives
the requestor abandoning it. New test
a_relayed_warm_and_a_local_warm_compete_for_the_same_slots fills every slot with relay-shaped claims
and asserts the local warm is refused, with a truthful control that it succeeds once a slot frees.
That direction is the one that matters; the reverse would be satisfied by any shared cap.

Finding 4 — orphaned doc

The block is back on warm_relayed, and holds has its own.


Also in this push

SPEC.md §11.1b — "verified complete" now names the asset's upload state, not name presence,
with the reason stated: the row exists at upload start, so all names can be present while bytes are
still in flight, and a name-counting reimplementation rebuilds the race #336 just fixed.

On 2a27fd3 — you are right, it is not an ancestor; the rebase rewrote it and I quoted the
pre-rebase hash. The substance stands: the ceiling was absent from four commits and is present and
correct at head.


Local evidence on the rebased tree (5fdcf909 base)

gate result
cargo fmt --all -- --check 0
cargo clippy --workspace --all-targets -- -D warnings 0
cargo test --workspace 0 — 973 in dig-node-core lib, 0 failed
revert-proofs 9 total, each run alone, each naming the assertion that fired

Still DRAFT. Re-gate requested.

Comment thread crates/dig-node-core/src/seams/dig_peer/module_transport.rs

@MichaelTaylor3d MichaelTaylor3d left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CHANGES-REQUIRED — scoped re-gate at 04494e09551daa00b0fdd2fe7604630270f225b9

(Independent fresh-context pass over the 6e2b673 -> 04494e0 delta. One gating finding, posted as an inline thread; everything else I was asked to check verified clean.)

Finding 1 (unification bypass) — VERIFIED, by reproduction

I reproduced the prior gate's bypass in my own worktree: replaced get_module_info's body with a direct self.ask_descriptor_once(..., false) and a match on the answer, deleting the ladder, the escalation and the relay wait from the shipped path.

Both production-path tests FAIL, and both fail bounded:

the_production_get_module_info_climbs_the_whole_ladder ... FAILED
  panicked at module_transport.rs:1649: `get_module_info` must be BOUNDED by the ladder ...: Elapsed(())
a_cold_first_invocation_escalates_through_the_production_transport ... FAILED
  assertion `left == right` failed: left: 1, right: 2

Restored, clean head is 25 passed; 0 failed for the module. So the tokio::time::timeout(ladder * 2, ...) wrapper does its job: the regression reports a red assertion in 0.01s, not a stuck job. The second test has no timer at all — its locator answers immediately and its dial path is synchronous-failing — so it is not hang-shaped in the first place. Both hang-shaped tests are bounded.

My own attempt at a NEW bypass:

  • descriptor_ask is a private mod and ask_within_deadlines is private to it. The only item the parent can reach that yields a descriptor is descriptor_for_holder, and it is pub(super) — nothing outside module_transport can reach any of it. No pub(crate), no re-export, no trait impl, no #[allow] opening a side door.
  • descriptor_over has exactly one caller (ask_descriptor_once), which has exactly one caller (the closure in get_module_info). Grepped crate-wide: no other call site.
  • The honest caveat: descriptor_over and ask_descriptor_once are private to the FILE, so a call site inside module_transport can still be written to skip the policy — which is precisely where the original bypass was written. So the defect is not compiler-unexpressible; it is test-detected, reliably, which I measured. I am satisfied with that, and the claim in the module doc is a shade stronger than what the privacy boundary alone delivers.
  • CountingLocator genuinely pins the CALL SITE, not a helper: the counter lives in find_providers, which is reached only via connect -> dial_targets -> discovered_candidates, once per dial, and a dial happens once per descriptor round. No helper increments it. Two rounds are producible only by the escalation, which only descriptor_for_holder implements.

Finding 2 (per-pull budget) — the arithmetic does NOT close. See the inline thread.

The within-one-pull arithmetic is right: remaining() is passed down as the ceiling, relay_waited is measured from immediately before wait_for_relayed_descriptor and is zero on every non-relaying branch, charge saturates, and the ceiling is checked before any further poll — so holders x 30 min no longer multiplies within a pull, and one honest hop can still spend the whole allowance.

What does not close is the ledger's LIFETIME. It is a field of a NatModuleTransport built once at node wiring (download.rs:2242), with no reset, no TTL and no clear on pull start or completion. So the key is the capsule for the life of the daemon, not the pull — and a capsule whose first honest pull spent the budget can never use the relay path again until restart, with the refusal attributed by name to a blameless peer. Detail, required fix, what the fix must not do, and the test it needs: the inline thread on module_transport.rs:329.

the_relay_budget_is_spent_per_capsule_and_saturates is a good test of what it covers, and it is structurally blind to this: it never models a second pull, so it passes identically with or without a reset.

Findings 3 and 4 — clear

Starvation is recorded at both ends and a_relayed_warm_and_a_local_warm_compete_for_the_same_slots carries the control arm (the local warm succeeds once a slot frees), so "refused" is not satisfied by a warm that never succeeds. Orphaned doc restored.

Rebase — four values read at HEAD, not from the diff

value read at 04494e0
root Cargo.toml version 0.146.0
dig-node-core version 0.55.0
lock dig-download 0.19.2
both chia-peer manifest comments present and naming dig_ecosystem#3152 (dig-node-core/Cargo.toml:115-122, dig-wallet/Cargo.toml:88-92)

The SPEC §11.1b addition says what it was claimed to say — verified-complete is the asset's upload state (state == "uploaded", vs "starting"), not the presence of a row.

Checks

check-merge-preconditions.sh asserts all five required contexts present and SUCCESS by name (Lint commit messages, Check version increment, Rustfmt, Clippy, Test + coverage). Its non-zero exit is the DRAFT flag only. build .msi is pending and non-required; not treated as blocking.

Not re-verified per the scoped brief: no-wait, both #332 revert-proofs, the NC-12 core claim, the §2.4b deferral behind #3152, the capsule_warm_locator_tests set assertion, the StagingDisposition justification.

Verdict recorded as a comment review because a same-identity PR 422s any verdict-bearing event. The open inline thread is what bars the merge.

RelayWaitBudget lived on a transport built once at node wiring, with no reset. So
the key was the capsule for the process lifetime: a capsule whose first honest pull
spent the budget could never take the relay path again until restart, and the
refusal was composed as a transport error naming a PEER -- blaming that peer for
this node's own earlier spend.

A TTL cannot fix it. A relay wait is charged when it ENDS, so a live wait looks idle
for up to the whole per-hop ceiling: any shorter TTL expires a live pull's budget
mid-wait and restores the holders x ceiling multiplication, and any longer one
blocks the next pull for as long as the harm it replaces. So the boundary is
reported rather than guessed -- PullLifecycle, called by the warmer that drives
download(), on every exit including failure.

Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d and others added 3 commits August 23, 2026 23:05
… cannot substitute

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

Re-gate round 2 addressed — ready for verdict

HEAD 26c11bd · base 5fdcf909 · still DRAFT, deliberately: this PR gates three epics, so it
merges on a verdict rather than on green.

Objective state

check-merge-preconditions.sh: mergeStateStatus=CLEAN, unresolvedReviewThreads=0, and all five
required contexts present and SUCCESS by name (Lint commit messages · Check version increment ·
Rustfmt · Clippy · Test + coverage). Local: fmt 0, clippy --workspace --all-targets -D warnings 0,
cargo test --workspace 0 — 976 in dig-node-core.

Values re-verified at HEAD after the rebase, by value rather than by "the file changed": root
0.146.0, dig-node-core 0.55.0, lock dig-download 0.19.2, #3152 named 4× in
dig-node-core/Cargo.toml and 1× in dig-wallet/Cargo.toml.

The two open items

Budget lifetime — fixed in 44409b1. PullLifecycle is reported by CapsuleWarmer immediately
after download() returns, before the outcome is inspected, so no early return can skip it;
NatModuleTransport releases that capsule's ledger entry. A TTL was considered and rejected for a
stated reason, recorded in the code: relay time is charged when a wait ends, so a live wait is
indistinguishable from an idle entry for up to the whole per-hop ceiling — any shorter TTL expires a
live pull's budget mid-wait and restores the multiplication, any longer one withholds the budget for
as long as the harm it replaces. Three tests, each revert-proved alone; the load-bearing one is
a_warmer_reports_the_end_of_every_pull_including_a_failed_one, because proving release correct says
nothing about whether anything calls it.

The over-claim — corrected in 26c11bd, in both places it appeared. The word "unbypassable" no
longer occurs in the file.

One note on a CI observation, so it is not mistaken for a defect

A dig-node-service --test never_log failure appeared in one local run with
os error 32 — the process cannot access the file because it is being used by another process. That is
a Windows file lock from two gate lanes running against dig-node worktrees concurrently, not a product
failure: re-run alone it passes 3/3, and it passed again in the full serialized run above.

Unchanged and still true

§2.6's bar is not cleared by this PR. Nobody has watched a .dig arrive over a hop on this branch; the
four timing constants are argued from the field measurements on #322/#332/#333, not re-measured. The
hardware acceptance run must go against this branch, since main's lock still resolves
dig-download 0.19.1.

@MichaelTaylor3d MichaelTaylor3d left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CHANGES-REQUIRED — scoped re-gate of 04494e0 -> 26c11bd

Head read: 26c11bdc2c4fdf877e650f2c1a3497c4269d9afb, resolved from the remote rather than taken from the dispatch. Base 5fdcf909. Scope: the two fixes only, per the re-gate brief.

One gating finding, posted inline on module_reshare.rs:802. Everything else in the delta verified and clears.

Fix 1 — budget lifetime: right shape, one uncovered exit path

  • Called on every returning path — verified. All four public entry points funnel into warm_with_config; one download call site, one pull_finished immediately after it; the three earlier returns sit above the download and have nothing to release.
  • Not called when the pull unwinds. See the inline finding: a panic inside download skips it, the tier-0 loop catches that panic by design (catch_iteration, a real catch_unwind), and the leaked entry reinstates both the permanent relay-ineligibility and the peer-named attribution the amended SPEC now forbids. A drop guard is strictly stronger and its absence matters.
  • Same object under both roles — verified. download.rs:2250 is the only non-test CapsuleWarmer::new, and it hands Arc::clone(&transport) as ModuleTransport and transport as PullLifecycle — one allocation, one ledger. No second transport is constructed on the production path.
  • The TTL rejection holds, premise checked. charge runs only after descriptor_for_holder returns (module_transport.rs:1512-1514), so nothing is recorded during a wait and a live entry is genuinely indistinguishable from an idle one for up to the whole ceiling. The argument is not merely plausible — its premise is how the code works.
  • Eviction cannot expire a live pull — discharged with a bound. WarmRegistry admits DEFAULT_MAX_CONCURRENT_WARMS = 4 distinct generations node-wide, relayed and local sharing the one cap, so at most four entries are ever live. charge evicts FIFO from the front only when inserting a new key at 1024 entries, and a live entry is always among the newest, so reaching one would require the 1023 older entries to be live too. Unreachable.

The three tests — re-run here, and they discriminate

test mutation applied result
a_warmer_reports_the_end_of_every_pull_including_a_failed_one pull_finished call deleted at module_reshare.rs:802, grep-confirmed applied RED in 0.00s
releasing_one_pull_does_not_refund_another release rewritten as a global clear() of both maps RED (left: 1800s, right: 900s)
a_second_pull_of_the_same_capsule_gets_its_own_budget same global-clear mutation green — confirming test 2 is what separates a scoped release from a reset

Baseline green in 0.04s before mutating. The third test's failing-pull half is load-bearing: under the mutation it is the failed-pull assertion at module_reshare.rs:1934 that fires first, so a fixture covering only the success path would not have caught it. Run in an isolated worktree, alone, per the os error 32 warning — no lock failure reproduced.

One process note on the re-run: filtering with cargo test ... --lib budget matches 27 tests and none of them is releasing_one_pull_does_not_refund_another, whose name contains no such substring. That filter reports ok while the test under examination never runs. Both budget-lifetime tests have to be named explicitly.

Fix 2 — the over-claim: correct

unbypassable no longer appears anywhere in the crate. The one remaining occurrence is SPEC.md:5972, a pre-existing and unrelated claim about an announcement backstop, byte-identical at 04494e0 and untouched here.

The replacement wording is accurate: Rust privacy is module-scoped, so the boundary is against the rest of the crate and not against an in-file edit, and the two tests it now leans on both exist and drive the real method — the_production_get_module_info_climbs_the_whole_ladder (module_transport.rs:1719) and a_cold_first_invocation_escalates_through_the_production_transport (:1784). The load-bearing claim is stated as detection rather than prevention, which is what is true.

Rebase values, read by value at HEAD

value at 26c11bd
root Cargo.toml version 0.146.0
dig-node-core version 0.55.0
Cargo.lock dig-download 0.19.2, registry source
#3152 in manifest comments present — root Cargo.toml:121; dig-node-core/Cargo.toml:356,394,404

Non-gating, resolved by me — eviction rationale is now stale

RelayWaitBudget's doc reads "evicting the oldest merely refreshes the budget of a capsule this node stopped working on long ago". After this fix, a capsule stopped working on long ago is no longer in the ledger at all — every entry is a live pull. The bound above makes it harmless, but the sentence argues for safety from a premise the fix removed. Worth one line when the file is next open; not a blocker, and not a reason for a round.

Scale note

This is a strict improvement on what is shipped today, where the entry leaks on every pull rather than only on a panicking one. The verdict is CHANGES-REQUIRED because the code and the normative SPEC now both assert a MUST the code does not hold, and the remedy is a local, roughly ten-line RAII change in the same file.

Shared-checkout disclosure

Read-only in the primary dig-node checkout: git fetch, git show, git diff, git grep, and git worktree add / prune. All probing happened in C:/tmp/worktrees/gate334, which is now removed. The primary working tree was not modified by me; the staged entries visible there pre-existed and belong to another lane.

Comment thread crates/dig-node-core/src/seams/dig_peer/module_reshare.rs Outdated
…nnot leak it

pull_finished was a plain statement after the download await, so an unwinding panic
skipped it. That is reachable rather than theoretical: the tier-0 precache path runs
the pull inside catch_iteration's catch_unwind (#2044), added so a panic there is
survived, and this crate does not build with panic = abort. The task then continued
with the ledger entry leaked -- restoring permanent relay-ineligibility for that
capsule AND the peer-naming exhaustion error that SPEC 21.1, as amended by this same
change, forbids.

PullBoundary reports on Drop, so the returning paths and the unwinding path go
through one mechanism. The eviction rationale is rewritten: with entries released per
pull and WarmRegistry capping live warms at 4 node-wide, the FIFO cap is a backstop
against a future leak, not the working mechanism.

Co-Authored-By: Claude <noreply@anthropic.com>
@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

Re-gate round 3 addressed — one item, fixed with a drop guard

HEAD 71f0de4 · base 5fdcf909 · DRAFT, awaiting verdict.

pull_finished was a plain statement after the download await, so an unwinding panic skipped it.
Reachable, not theoretical: the tier-0 precache path runs the pull inside catch_iteration's
catch_unwind — added by #2044 so that a panic there is survived — and this crate does not build
with panic = "abort". PullBoundary now reports on Drop, scoped tightly around the download, so
the returning paths and the unwinding path go through one mechanism.

a_pull_that_panics_still_reports_its_end drives a panicking holder through the real warm entry
point and through the same AssertUnwindSafe(...).catch_unwind() the tier-0 loop uses. Restoring the
plain statement turns exactly that test red while the other 23 in the module stay green.

Stale eviction prose rewritten: with per-pull release and WarmRegistry capping live warms at 4
node-wide, the FIFO cap is a backstop against a future leak, not the working mechanism.

A verification note worth recording on the PR, not just in the thread

The gate warned that cargo test --lib budget matches 27 tests and misses
releasing_one_pull_does_not_refund_another. The --exact form is worse. cargo test --lib <bare_name> -- --exact matches nothing, because --exact wants the full module path — all six of
my budget/lifecycle tests reported ok. 0 passed before I checked the count. With full paths
(seams::dig_peer::module_transport::tests::…) each reports 1 passed.

A green from a filter that matched nothing is the same failure class as a green from a fixture that
cannot reach the code under test: the suite says ok and nothing ran.

State

mergeStateStatus=CLEAN · unresolvedReviewThreads=0 · all five required contexts SUCCESS by
name
. Local: fmt 0, clippy --workspace --all-targets -D warnings 0, cargo test --workspace 0 —
977 in dig-node-core.

Unchanged

§2.6's bar is still not cleared by this PR: nobody has watched a .dig arrive over a hop on this
branch, and the four timing constants are argued from the field measurements on #322/#332/#333 rather
than re-measured. The hardware acceptance run must go against this branch — main's lock still
resolves dig-download 0.19.1.

@MichaelTaylor3d MichaelTaylor3d left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: PASS — re-gate round 3 (scoped)

Head reviewed: 71f0de49656b680fa565577218e662f26a0875bb (base 5fdcf909; delta from 26c11bd = 2 files, +151/-8).
Recorded as a --comment review: a same-identity PR 422s any verdict-bearing event, so this comment IS the verdict.

Scope was the single PullBoundary fix and its test. Everything cleared in rounds 1-2 was not re-verified.

1. The guard is a strict superset, not a re-shuffle

module_reshare.rs:844-847. Exactly ONE pull_finished call site existed before (the plain statement after the
.await); exactly ONE mechanism exists now (Drop for PullBoundary), and the drop point is the SAME position —
the block closes immediately after the await, with nothing between it and the old statement's line.

  • Every returning path (Ok, Err) and the unwinding path now go through one mechanism.
  • Nothing that previously reported now does not. Paths that return BEFORE the download (no chain anchor, warm-cap
    skip, non-canonical id) did not report before either, and correctly still do not — no relay budget has been
    charged at that point.
  • Strictly better on a third path the old code missed: future CANCELLATION. If the warm future is dropped
    mid-download the guard still runs; the plain statement did not.

2. Scope is right

Verified the code between the block and the old call site: StagingDisposition::for_failure / apply,
promote_into_cache, pull.erase(), announce_inventory(), and the tracing calls. None of them touches
ModuleTransport or charges RelayWaitBudget under (store_id, root), so releasing at block-end rather than at
end of function relies on nothing still-live.

3. The guard cannot panic, and cannot be a silent no-op

Drop::drop -> PullLifecycle::pull_finished -> NatModuleTransport::release (module_transport.rs:379-393).
That path is:

  • Poison-tolerant on both mutexeslock().unwrap_or_else(|p| p.into_inner()), so a panic that poisoned the
    budget mid-charge cannot make the release panic during a subsequent unwind. No unwrap, no expect, no ?.
  • Lock-order consistent with charge (spent then order in both), so no inversion deadlock.
  • No I/O, no logging, no fallible allocation beyond format! — nothing that can panic during an unwind and
    turn a survivable panic into a process abort.

pull_lifecycle is Arc<dyn PullLifecycle>, never an Option, so as_ref() is infallible and the guard is never
absent. The only no-op impl, NoPullState, is #[cfg(test)] and its doc comment says why — production has one
wiring and it is NatModuleTransport.

4. No second download call site

At 71f0de4 there is exactly one production ModuleDownloader::download in the crate
(module_reshare.rs:846), inside the boundary. The other .download( hit in the tree is
download.rs:2044, a different type on the content-id path that does not carry the relay budget. The delta added
none.

The revert-proof, re-run independently — passed-count read explicitly

Own detached worktree at 71f0de4 (C:/tmp/worktrees/gate-334-rp); the shared checkout and the lane's own
dn-3128 worktree were never touched.

Baseline: test result: ok. 24 passed; 0 failed — 24, not 23; the module has 24 including the new test.

Mutation — restored the plain statement, grep-confirmed BEFORE running: zero occurrences of
PullBoundary::new, one occurrence of let pulled = downloader.download(...) at line 844. No cargo fmt was run,
so the mutation could not be silently reformatted away.

Mutated: test result: FAILED. 23 passed; 1 failed — and the one failure is exactly
a_pull_that_panics_still_reports_its_end, on the intended assertion, with left: [] against the expected
[(store, root)] pair.

Three things this establishes, beyond "it went red":

  • The control is real and fired FIRST. outcome.is_err() passed under the mutation, so the pull genuinely
    panicked and was genuinely caught. "Reported its end" could not have been satisfied by a pull that never panicked.
  • The test names the PROPERTY, not an outcome. It failed with an ABSENT report, which is a placement failure.
    The nearest wrong implementation — report after the await — is precisely what was restored, and it does not pass.
  • No other test depends on the guard; the other 23 stayed green, so the fix is not load-bearing anywhere it
    was not meant to be.

Worktree restored to clean (git status --porcelain empty) after the run.

Non-ASCII sweep of the delta

Swept every ADDED line across the full PR delta (5fdcf909..71f0de4), not just the two files this round touched.
Surviving non-ASCII, complete census: 80 em-dash, 6 multiplication-sign, 4 section-sign. All deliberate prose.
Zero CJK / kana / hangul — the two stray characters are gone and nothing similar replaced them.

Stale eviction prose

module_transport.rs:330-340 now states the true mechanism: entries released per pull via PullLifecycle,
WarmRegistry capping live warms node-wide, and the 1024 FIFO cap as a bound on a FUTURE leak rather than the
working mechanism. Confirmed DEFAULT_MAX_CONCURRENT_WARMS = 4 at module_reshare.rs:64, so the reasoning it
rests on holds by value.

Rebase values at HEAD, by value

  • root Cargo.toml version = 0.146.0
  • crates/dig-node-core/Cargo.toml version = 0.55.0
  • Cargo.lock dig-download = 0.19.2 (crates.io registry source)
  • dig_ecosystem#3152 named in the manifest comments — 4 occurrences across root + core, each explaining a
    deliberate hold on the chia-0.36 line

Diff is 14 files, +1999/-271, as stated.

Non-gating notes (raised and self-resolved, not blocking)

  1. Both new assertion messages carry an interior run of spaces from a wrapped string literal, so the failure text
    reads with a gap. Cosmetic, appears only on failure output. Not worth a round.
  2. PullBoundary borrows rather than owning, so it cannot outlive the warmer — correct today, and worth
    remembering if the pull is ever moved into a spawned task, where an owned Arc would be needed instead.

No gating findings. No threads opened.

@MichaelTaylor3d
MichaelTaylor3d marked this pull request as ready for review August 24, 2026 10:27
@MichaelTaylor3d
MichaelTaylor3d merged commit 7b4caab into main Aug 24, 2026
15 checks passed
@MichaelTaylor3d
MichaelTaylor3d deleted the loop/3128-relay-batch branch August 24, 2026 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant